Accelerating the assessment agenda: thinking outside the black box

Over the last 10 years, learning and teaching in higher education have benefited from advances in social constructivist and situated learning research (Laurillard, 1993). In contrast, assessment has remained largely transmission orientated in both conception and in practice (see Knight & Yorke, 2003). This paper examines a number of recent developments, which exhibit innovation in electronic assessment developed at the UK's Open University. This paper argues for the development of new forms of e-assessment where the main driver is that of sound pedagogy rather than state of the art technological know-how and where open source products can move the field forward

Computer assisted formative assessment: supporting students to become more reflective learners

e-Assessment is being advocated in the UK as our way of introducing a more personalised learning agenda throughout the Higher Education sector. This paper discusses the findings from two projects where formative e-assessment has contributed to students taking more control of their own learning. One study set out to provide further insights into the role of electronic formative assessment and to point the way forward to new assessment practices, capitalising on a range of open source tools. The guiding vision was to pilot a series of formative assessments which have the potential to help shape learners as independent thinkers, making their own judgements and decisions about their learning in partnership with their peers and tutors. Other work consisted of evaluating a series of formative assessments given to Philosophy students. Lessons have been learned about the type of feedback that instructors and students think will be most useful and how using theis type of application promotes self reflection. The research reported here starts to illustrate how technology can be adapted to become more 'fit for pedagogical purpose?. The feedback offered by these systems encourages learner metacognition and aims to empower students to reflect and become independent thinkers. This approach sits well within a constructivist paradigm which has often been less well served in the past through formal summative assessment which is not an integral part of the knowledge construction process

Developing a Roadmap for e-Assessment: Which Way Now?

e-Assessment is of strategic importance to the UK since it forms an integral part of the e-learning movement which is a major global growth industry. This paper reports results from a project commissioned by JISC which set out to develop a Roadmap for e-assessment. This methodological approach was drawn from a range of 'roadmap? methodologies collected by Glenn and Gordon (2003). It facilitated the identification of the enabling factors and barriers to the use e-assessment through the construction of a survey which probed a number of experts opinions. The analysis of the various sources suggest that in England and Wales it is policy pressure which is a main driver and it is affecting more of the FE sector than the HE sector. In the HE sector institutions have more control over the rate and uptake of e-assessment as they award their own degrees. However, there is a recognition in HE that with larger classes and less tutorial time, tutors can keep track of their students? progress through e-assessment systems. They can adjust their lectures accordingly after they have picked up the misconceptions of a cohort through e-assessment feedback. At a personal level teachers/enthusiasts are addressing pedagogical problems through e-assessment. The barriers identified at a superinstitutional level, for example the . DfES, funding bodies, and examining bodies, are that of regulation, confidentiality and testing of these systems before they go across the UK. While the main drivers at a superinstitutional level are to move towards a new generation of learners engaged in self-reflection who will be able to identify their own learning needs. One of the major drivers for institutions to adopt e-assessment practices is that of student retention. HE and FE also see benefits with respect to attendance and achievement. This paper outlines the methods used and describes key barriers which will have to be overcome if e-Assesment is to be effectively deployed across UK HE and FE sectors

Identifying innovative and effective practice in e-assessment: findings from seventeen UK case studies

The aim of this JISC funded project was to extend the understanding of what e-assessment meant to users and producers in the HE and FE sectors. A case study methodology was employed to identify and report upon best and current practice within this field of inquiry. This approach facilitated the identification of both the enabling factors and barriers associated with e-assessment. The variety of applications of e-assessment studied and their innovation and general effectiveness indicate the potential of e-assessment to significantly enhance the learning environment and the outcomes for students, in a wide range of disciplines and applications

Improving a tutor’s feedback assessment tool: transforming Open Mentor following two recent deployments

Evidence shows the vital role that the quality of feedback plays on students’ performance and on the overall increase of learning opportunities that good feedback creates for students. Based on this evidence, the Open University developed Open Mentor (OM), a system to support tutors enhance their feedback practice. Open Mentor Technology transfer (OMTetra), a JISC funded project, took OM and deployed it in two Higher Education institutions with the purpose of evaluating the process of transferability and continue the development of the tools available to tutors within the system. This paper describes the original OM and the enhancements identified after use and evaluations from tutors of the institutions involved

Addressing the Challenges of Assessment and Feedback in Higher Education: A collaborative effort across three UK Universities

Assessment has been identified as one of the major challenges faced by Higher Education Institutions (Whitelock, et al, 2007). As a response to the challenge, in a project funded by the Joint Information Systems Committee (JISC) Open Mentor (OM) was developed as a learning support tool for tutors to help them reflect on the quality of feedback given to their students on assignments submitted electronically. Its development was based on the fundamental theory that there was convincing evidence of systematic connections between different types of tutor comments and the level of attainment in an assignment (Whitelock, et al 2004). OM analyses, filters, and classifies tutor comments through an algorithm based on Bale’s Interaction Process. As a result, tutor’s feedback comments are classified into four categories namely: Positive reactions, Teaching points, Questions and Negative reactions. The feedback provided is analysed against an ideal number of feedback comments that an assignment given a mark of a specific band should have. Reports are provided in OM to support tutors in the task of reflecting on their feedback structure, content and style. The JISC-funded Open Mentor technology transfer (OMtetra) project is continuing the work initiated by the Open University implementing OM at the University of Southampton and King’s College London. OMtetra aims at taking up OM and extending its use by developing the system further and ultimately offering better support to tutors and students in the assessment process. A group of tutors from the University of Southampton and Kings’ College are at present using OM in their teaching and assessment. In this paper, we explore potential improvements to OM in three aspects: user interface, technology implementation and analysis algorithm design. For the user experience aspect suggested additions to OM include the creation of a simple entry form where tutors may validate the results of the analysis of the feedback comments. In addition, enhancements to OM will facilitate uploading of students and modules information into the system. Presently, OM utilises a built-in database of users that needs to be maintained separately from institutional systems. Improvements for this system feature include a more flexible authentication module which would simplify the deployment of the system in new environments and thus promote uptake by a larger number of institutions. In order to reach this goal, the system will be migrated to an open source framework which provides out-of-the-box integration with various authentication systems. The last to improve is the analysis algorithm. Currently, OM classifies tutors’ comments into four categories by applying an underlying text matching algorithm. This method could be improved if tutors are allowed to confirm comments’ classification through the OM interface and a free-text classification algorithm. As the number of users grow, so will the algorithm and analysis process, making it more comprehensive and intelligent as the keywords used during analysis are dynamically expanded. OMtetra is an on-going project with a lot of potential. We believe that the outcomes from the development and trial implementations of OM will contribute highly to the area of assessment in higher education

An embedded approach to plagiarism detection using the TeSLA e-authentication system

Plagiarism continues to remain an ever-present issue throughout academia. It is an anathema to scholarly enterprise, where the proper attribution of the work of others is of fundamental importance. Teaching students the importance of citing and referencing the work of others, and how to correctly do so, is therefore an important role for academic institutions. It is insufficient to teach these things without assessing students’ learning. Effective and accessible tools that can assist in teaching and assessment are sought and are increasingly being developed. This paper describes a new tool designed to assess levels of plagiarism in students’ submitted work and considers its affordances alongside other established tools. TeSLA is an EU funded project that brings eighteen partners together for the development of an embedded suite of tools to deliver the seamless e-authentication of students as they complete online assessments. Within the suite is a plagiarism detection tool that analyses documents and text on submission and provides immediate output. We show that the TeSLA plagiarism detection tool highlights potential collusion, a form of plagiarism. Also, we discuss whether the embedded nature of the TeSLA system could be used to improve constructive alignment between teaching and assessment within modules

Overcoming cross-cultural group work tensions: mixed student perspectives on the role of social relationships

As universities worldwide rapidly internationalise, higher education classrooms have become unique spaces for collaboration between students from different countries. One common way to encourage collaboration between diverse peers is through group work. However, previous research has highlighted that cross-cultural group work can be challenging and has hinted at potential social tensions. To understand this notion better, we have used robust quantitative tools in this study to select 20 participants from a larger classroom of 860 students to take part in an in-depth qualitative interview about cross-cultural group work experiences. Participant views on social tensions in cross-cultural group work were elicited using a unique mediating artefact method to encourage reflection and in-depth discussion. In our analysis of emergent interview themes, we compared student perspectives on the role of social relationships in group work by their academic performance level. Our findings indicated that all students interviewed desired the opportunity to form social relationships with their group work members, but their motivations for doing so varied widely by academic performance level

The negotiation and co-construction of meaning and understanding within a postgraduate online learning community

There is an increasing development of courses and course components taught through teaching and learning dialogues online yet there is little secure knowledge regarding the educational quality and outcomes of these dialogues. Drawing on contemporary socio-cultural research, this paper adapts a well-established analytical framework (see Mercer, 1995) that has been developed to understand face to face educational dialogues to the new context of asynchronous electronic conferencing. The work reported is derived from an in-depth case study of a tutorial group of 11 students enrolled on a course within the Open University's MA in Open and Distance Learning. The course was taught on-line to an international cohort of students from wide-ranging academic backgrounds. The analyses of electronic conference archives presented here focus on understanding the students’ on-line collaborative work and the ways in which they constructed meaning, negotiated shared understanding and supported each other in the process of learning at a distance. The implications of the findings for educational practice are considered

Eating disorders in weight-related therapy (EDIT): protocol for a systematic review with individual participant data meta-analysis of eating disorder risk in behavioural weight management

The Eating Disorders In weight-related Therapy (EDIT) Collaboration brings together data from randomised controlled trials of behavioural weight management interventions to identify individual participant risk factors and intervention strategies that contribute to eating disorder risk. We present a protocol for a systematic review and individual participant data (IPD) meta-analysis which aims to identify participants at risk of developing eating disorders, or related symptoms, during or after weight management interventions conducted in adolescents or adults with overweight or obesity. We systematically searched four databases up to March 2022 and clinical trials registries to May 2022 to identify randomised controlled trials of weight management interventions conducted in adolescents or adults with overweight or obesity that measured eating disorder risk at pre- and post-intervention or follow-up. Authors from eligible trials have been invited to share their deidentified IPD. Two IPD meta-analyses will be conducted. The first IPD meta-analysis aims to examine participant level factors associated with a change in eating disorder scores during and following a weight management intervention. To do this we will examine baseline variables that predict change in eating disorder risk within intervention arms. The second IPD meta-analysis aims to assess whether there are participant level factors that predict whether participation in an intervention is more or less likely than no intervention to lead to a change in eating disorder risk. To do this, we will examine if there are differences in predictors of eating disorder risk between intervention and no-treatment control arms. The primary outcome will be a standardised mean difference in global eating disorder score from baseline to immediately post-intervention and at 6- and 12- months follow-up. Identifying participant level risk factors predicting eating disorder risk will inform screening and monitoring protocols to allow early identification and intervention for those at risk